The present invention relates to the field of displaying high resolution images on displays with lower resolution.
The most commonly used method for displaying high-resolution images on a lower resolution color mosaic display is to prefilter and re-sample the pixels 2 of the high-resolution image 4 down to the resolution of the low-resolution display 6, as shown in FIG. 1. In the process, the R, G, B values of selected color pixels 8 are mapped to the separate R, G, B elements 10, 12 and 14 of each display pixel 16. These R, G, B elements 10, 12 and 14 of a display pixel are sometimes also referred to as subpixels. Because the display device does not allow overlapping color elements, the subpixels can only take on one of the three R, G, or B colors. The color's amplitude, however, can be varied throughout the entire grey scale range (e.g., 0–255). The subpixels often have a 1:3 aspect ratio (width:height), so that the resulting pixel 16 is square. The aforementioned subsampling/mapping techniques fail to consider the fact that the display's R, G, and B subpixels are spatially displaced; in fact the pixels of the low resolution image are assumed to be overlapping in the same manner as they are in the high-resolution image. This type of sampling may be referred to as sub-sampling, traditional sub-sampling, or ordinary sub-sampling.
The pixels of the high-resolution image 4 are shown as three slightly offset stacked squares 8 to indicate their RGB values are associated for the same spatial position (i.e., pixel), generally referred to as co-sited sub-pixels. One display pixel 16 on a color mosaic display, consisting of one each of the R, G and B subpixels 10, 12 and 14 is shown as part of the lower-resolution triad display 6 in FIG. 1.
In the example shown in FIG. 1, the high-resolution image has 3× more resolution than the display (in both horizontal and vertical dimensions). In the case that filtering is omitted, the subsampling process would cause undesirable aliasing artifacts, and, accordingly, various methods are used, such as averaging the neighboring un-sampled pixels in with the sampled pixel, to reduce the aliasing. In addition, the subsampling technique of FIG. 1 results in mis-registration of the color fields each of which carries a portion of the luminance information. This leads to a loss of luminance resolution attainable at the sub-pixel sampling rate.
It is noted that the technique of weighted averaging of neighboring elements while subsampling is mathematically equivalent to prefiltering the high resolution image. Also, it is noted that techniques of selecting a different pixel than the leftmost (as shown in FIG. 1) can be considered as a prefiltering that affects only phase. Thus, most of the processing associated with reducing aliasing may be viewed as a filtering operation on the high-resolution image, even if the kernel is applied only at the sampled pixel positions, or both.
It has been realized that the aforementioned techniques do not take advantage of potential display resolution. Information regarding potential display resolution is discussed by R. Fiegenblatt (1989), “Full color imaging on amplitude color mosaic displays” Proc. SPIE V. 1075, 199–205; and J. Kranz and L. Silverstein (1990) “Color matrix display image quality: The effects of luminance and spatial sampling,” SID Symp. Digest 29–32, incorporated herein by reference.
For example, in the display shown in FIG. 1, while the display pixel 16 resolution is ⅓ that of the pixel resolution of the high resolution image (source image) 4, the subpixels 10, 12 and 14 of the low resolution image are at a resolution equal to that of the high resolution image (in the horizontal dimension). This may be taken advantage of as shown in FIG. 2. In the case that the low resolution display were to be viewed solely by a color blind individual, he would see it as a higher resolution image than if ordinary sub-sampling is used. In essence, a luminance value exists for each pixel of the high resolution image which is mapped to a corresponding sub-pixel of the low resolution image. In this manner, a portion of the high resolution luminance image 4 is preserved in the sub-pixels of the low resolution image. This approach is shown in FIG. 2, where the R, G, and B subpixels 10, 12 and 14 of the low resolution display are taken from the corresponding colors of different pixels 11, 13 and 15 of the high-resolution image. This allows the sub-pixel horizontal resolution of the low resolution display to be at the pixel resolution of the high resolution display. Sampling which comprises mapping of color elements from different image pixels to the subpixels of a display pixel triad may be referred to as sub-pixel sampling.
But what about the viewer of the display who is not color-blind? That is, the majority of viewers. Fortunately for display engineers, even observers with perfect color vision are generally color blind at the highest spatial frequencies. This is indicated in FIG. 3, where idealized spatial frequency responses of the human visual system are shown.
In FIG. 3, luminance Contrast Sensitivity Function (CSF) 17 refers to the achromatic content of the viewed image, and chrominance CSF 19 refers to the color content, which is processed by the visual system as isoluminant modulations from red to green, and from blue to yellow. The color difference signals R-Y and B-Y of typical video are rough approximations to these modulations. For most observers, the bandwidth of the chromatic frequency response is ½ that of the luminance frequency response. Sometimes, the bandwidth of the blue-yellow modulation response is even less, down to about ⅓ of the luminance.
With reference to FIG. 4, in the horizontal direction of the display, there is a range of frequencies that lie between the Nyquist frequency of the display pixels 16 (display pixel=triad pixel, giving a triad Nyquist at 0.5 cycles per triad pixel) and the Nyquist frequency of the sub-pixels 10, 12 and 14 (0.5 cycles per subpixel=1.5 cycles/triad pixels). This region of frequencies is shown as the rectangular region 20 in FIG. 4. The result of convolving the high resolution image with a rect function whose width is equal to the display sample spacing is shown as a dashed-dot curve 22. This is the most common approach taken for modeling the display MTF (modulation transfer function) when the display is a LCD.
The result of convolving the high-res source image with a rect function whose width is equal to the subpixel spacing is shown as a dashed curve 24, which has higher bandwidth. This is the limit imposed by the display considering that the subpixels are rect in ID. In the shown rectangular region 20, the subpixels can display luminance information, but not chromatic information. In fact, any chromatic information in this region is aliased. Thus, in this region, by allowing chromatic aliasing, the display may achieve higher frequency luminance information than allowed by the triad (i.e., display) pixels. This is the “advantage” region afforded by using sub-pixel sampling.
The sub-pixel sampling registers the luminance information in the three color fields of the displayed image. Mis-registration as a result of displaying the image causes loss of luminance resolution while sub-pixel sub-sampling reduces it. The sub-sampling prefilter applied to the image may be sufficiently broad to permit the high resolution luminance information to pass. This additional luminance resolution will not result in significant aliasing of the luminance information because the Nyquist frequency is determined by the sub-pixel sampling period. However, significant chromatic aliasing can occur because the chromatic Nyquist frequency is determined by the display sampling period. The “advantage” region may be thought of as where significant chromatic aliasing occurs and significant luminance aliasing does not occur.
For applications with font display, the black and white fonts are typically preprocessed, as shown in FIG. 5. The standard pre-processing includes hinting, which refers to the centering of the font strokes on the center of the pixel, i.e., a font-stroke specific phase shift. This is usually followed by low-pass filtering, also referred to as grey scale anti-aliasing.
The visual frequency responses (CSFs) shown in FIG. 3 are idealized. In practice, they have a finite falloff slope, more representatively shown in FIG. 6A. The luminance CSF 30 has been mapped from units of cy/deg to the display pixel domain (assuming a viewing distance of 1280 pixels). It is shown as the solid line 30 that has a maximum frequency near 1.5 cy/pixel (display pixel), and is bandpass in shape with a peak near 0.2 cy/pixel triad. The R:G CSF 32 is shown as the dashed line, that is lowpass with a maximum frequency near 0.5 cy/pixel. The B:Y CSF 34 is shown as the long dashed LPF curve with a maximum frequency similar to the R:G CSF, but with lower peak response. The range between the cutoff frequencies of the chroma CSF 32 and 34 and the luminance CSF 30 is the region where one may allow chromatic aliasing in order to improve luminance resolution. The chromatic aliasing will not be visible to the human eye because it falls outside the chromance CSF.
FIG. 6A also shows an idealized image power spectra 36 as a 1/f function, appearing in the figure as a straight line with a slope of −1 (since the figure is using log axes). This spectrum will repeat at the sampling frequency. The pixel repeat 38 is due to the pixel sampling rate, and the repeat 40 is due to the subpixel sampling rate. Note that the shapes of the repeat spectra are different than the 1/f base band spectra 36, because they are plotted on log-log axes. The portions of these repeat spectra 38 and 40 that extend below their respective Nyquist frequencies represent aliasing. The leftmost one is chromatic aliasing 38 since it is due to the pixel sampling rate, while the luminance aliasing 40 occurs at higher frequencies because it is related to the higher sub-pixel sampling rate.
In FIG. 6A, no prefiltering has been applied to the source spectra. Consequently, aliasing, due to the pixel sampling (i.e., chromatic aliasing), extends to very low frequencies 35. Thus even though the chromatic CSF has a lower bandwidth than the luminance CSF, the color artifacts will, in general, still be visible (depending on the noise and contrast of the display).
In FIG. 6B, a prefilter was applied (a rect function in the spatial domain equal to three source image pixels), shown in FIG. 4 as a dashed-dotted line 22, to the source power spectrum, and it affects the baseband spectrum 42 in the region of 0.5 cy/pixel and greater, causing it to have a slope steeper than −1 shown at 44. The steeper slope effectively reduces the effects of the chromatic aliasing. The repeat spectra 38a and 40a also show the effect of this prefilter. For example, the tail 35 (FIG. 6A) is dramatically reduced as tail 46 (FIG. 6B) with this filter. The visible chromatic aliasing, that is aliasing under the two chrominance CSFs 32a and 34a, is reduced. However, it can be observed that this simple luminance prefiltering also removes significant luminance resolution (e.g. the curve 44 (FIG. 6B) relative to curve 45 (FIG. 6A)).
To increase the luminance information a system may use the difference in the human visual system's luminance and chrominance bandwidth. This bandwidth difference in luminance and chrominance (CFSs) in FIG. 6B may be referred to as the “advantage region”. One technique to achieve such a boost is to design the prefiltering based on visual system models as described in C. Betrisey, et al (2000), “Displaced filtering for patterned displays,” SID Symposium digest, 296–299, incorporated by reference and illustrated in FIG. 7.
The Betrisey, et al. technique ideally uses different prefilters depending on which color layer, and on which color subpixel the image is being sampled for. There are 9 filters designed using a human visual differences model described in Zhang and B. Wandell (1996) “A spatial extension of CIELAB for digital color image reproduction,” SID Symp. Digest 731–734, incorporated herein by reference and shown in FIG. 7. This was done offline, assuming the image is always black and white. In the final implementation, three rect functions rather than the resulting nine optimal filters are used in order to save computations. In addition, there is still some residual chromatic error that can be seen because the chromatic aliasing extends down to lower frequencies than the chromatic CSF cutoff (as seen in FIG. 6B).